Stuff I learned while writing a Linux course

Codemonkeys 04/08/16

Jupyter with a bash kernel

Install from here:

https://github.com/takluyver/bash_kernel

works as described:



In [3]:

    
ls /var









    



backups  crash  local  log   metrics  run    tmp
cache    lib    lock   mail  opt      spool

`tree` outputs directory structures to the terminal



In [5]:

    
# first apt-get install tree
tree -L 2 -h --du ~/Dropbox/Code/Axelrod_stuff/Axelrod









    



/home/martin/Dropbox/Code/Axelrod_stuff/Axelrod
├── [110K]  axelrod
│   ├── [ 249]  actions.py
│   ├── [4.3K]  deterministic_cache.py
│   ├── [3.0K]  ecosystem.py
│   ├── [2.2K]  eigen.py
│   ├── [ 950]  game.py
│   ├── [ 643]  __init__.py
│   ├── [3.3K]  interaction_utils.py
│   ├── [4.8K]  match_generator.py
│   ├── [4.8K]  match.py
│   ├── [1.4K]  mock_player.py
│   ├── [4.3K]  moran.py
│   ├── [4.8K]  player.py
│   ├── [9.6K]  plot.py
│   ├── [ 528]  random_.py
│   ├── [ 26K]  result_set.py
│   ├── [4.0K]  strategies
│   ├── [ 11K]  strategy_transformers.py
│   ├── [4.2K]  _strategy_utils.py
│   ├── [4.0K]  tests
│   └── [ 12K]  tournament.py
├── [1.9K]  CHANGES.txt
├── [ 644]  CITATION.rst
├── [ 33K]  docs
│   ├── [ 333]  citing_the_library.rst
│   ├── [8.4K]  conf.py
│   ├── [1.9K]  index.rst
│   ├── [6.6K]  Makefile
│   ├── [4.0K]  reference
│   ├── [   0]  requirements.txt
│   ├── [4.0K]  _static
│   └── [4.0K]  tutorials
├── [ 333]  doctests.py
├── [1.1K]  LICENSE.txt
├── [  43]  MANIFEST.in
├── [6.0K]  README.rst
├── [  88]  requirements.txt
├── [ 578]  setup.py
└── [1.6K]  test

 159K used in 7 directories, 32 files

There's a `-t` threshold option for du



In [12]:

    
#rather than this...
du -h --max-depth 2 ~ | grep G









    



12G	/home/martin/.local/share
12G	/home/martin/.local
7.7G	/home/martin/Dropbox/Photos
11G	/home/martin/Dropbox/Jane's dropbox
25G	/home/martin/Dropbox/archived
5.9G	/home/martin/Dropbox/projects
4.1G	/home/martin/Dropbox/Code
4.4G	/home/martin/Dropbox/downloads
3.7G	/home/martin/Dropbox/python_for_biologists
1.5G	/home/martin/Dropbox/science
2.1G	/home/martin/Dropbox/Camera Uploads
66G	/home/martin/Dropbox
1.3G	/home/martin/Desktop/files
1.5G	/home/martin/Desktop
2.1G	/home/martin/.steam/steam
312K	/home/martin/.steam/WINDOWSTEMPDIR_FONTCONFIG_CACHE
1.2G	/home/martin/.steam/ubuntu12_32
3.8G	/home/martin/.steam
2.5G	/home/martin/Downloads/get_iplayer-2.94
6.5G	/home/martin/Downloads
94G	/home/martin



In [14]:

    
# ...do this
du -h --max-depth 2 -t 5G ~









    



12G	/home/martin/.local/share
12G	/home/martin/.local
7.7G	/home/martin/Dropbox/Photos
11G	/home/martin/Dropbox/Jane's dropbox
25G	/home/martin/Dropbox/archived
5.9G	/home/martin/Dropbox/projects
66G	/home/martin/Dropbox
6.5G	/home/martin/Downloads
94G	/home/martin

You can get crude memory benchmarking with `/usr/bin/time -v`

For some reason that I don't fully understand, you have to give the full path to time for this to work:



In [15]:

    
# how much memory does it take to calculate the sum of the first twenty thousand square numbers in python?
/usr/bin/time -v python -c 'print(sum([2 ** x for x in range(20000)]))'









    



398027684033796659235430720619120245370477278049242593871342686565238635974930057042676009749975595510836461137504912702831400376935319143621753470415827025981215282426893498224826615977707595539466961019588699726772279731941315198182787264034852821200164566127930390710398182979935327718016873784821349516406114982916691867361875370024545872140793827277482562824192439237801588697814168520338650090909697535966525032757049430286459482977357373598020450589927318365663076719136934132593126761906696003770385305284570331119691001526584347722012386381881779425549210851696458253943578557699072154639655630793883941961378971846841113804188730258903839103669626086974468150655710480841592465655211805257863007811676888839555017536731758113448656752514158601444051645154665514388431619042396106716755762338728183461369854648923972904427556158821823778729193111453445844216979095435045778144571378954652122396061615147642540250745857228893999875491625014946013839340891326060933901036249999238637827577774666644809734033861619420363936465178730919233673114244563915058438996625834112132967998495576249320462871747777012165543887156255858358784852335060574881876552025685704823768078710818951860741379429242110855644973977420413810373514584504006896392675854997866870818564207239083874324953871276375716101506575153205747363963740749867514682619756775534507006871485887812402927738227576635284174246988540785975240020481266853076127172228024330561550120182008777598230542033702463408316671120886169260934006805799864598636311179787776738608992346063063099659648279663878174074787179237169752957046404584525301384153358344055908219695854852185210739761460551596658211013159915409566145426809737550417578228465835830890294497535463112081537672664056891624345779311524560019984315456142126282898486728345004767873499752683471409587367450593302392307908004590644754012537113320493601682133709318222647489080531644015321391157387178232154126828007760313716872242209614200967522180475716199973689467714010404673961454146466045855232217196687665143147612199151921277432309700460321430381533385245877431330533479476152339364503436322919665631042328740463612565842560411947020174006507893396276103834436233140915025391014386119201176462659556388343058600326710618903683746516577021214276933289179021059956925949717956040857979165914170970056212869933593589268626151996676594370800885093048230687152803213254735594741799076039453057272319884322341883241036382617598401889439130301876975498681736174215711287053447013711596004574803562701388246822510391522419061320663740921321754344166744899588160649291823535983386025904942040724581017615968429577015808090360968544059204594200069304612417366398776831532265596224715750301792207725607932534543693758772262010387360435567635232718343420679693057360004073679493008945813961012439574397373178636054628207647520675194420244271036343729318858430871461978866964772362057290577326080664463129657590249859748544101333842092713653096656066266827446079145590196644643417403723220085696202719321533233027169599734928971588850348415000070034027025298183104148343980297663148971586607903771717880683175436445585810610546882073571556162324659351310326560804448974229349743425637164834242799991427145050899469511954834774847172360693568437689147399455672090773686782511054291185172381917008889957645311339950993044779783607140593766508017935992581357858306525303783231752425242008347844867988333025417249944092118578113687403158162707075154006053416374075765162668533127078605316562826337193606242535290683224423660462222408680300498714149607265550441220738075941633988435051594487256802874182264814425923111193188280632013127802897889605338783089532740877202304122498193625454768343775535498872821099981620497070810489137457106892573248498734243717184800822956334469415666818858073218653977954309023182851723246522042792401461382001601920501284439325214084210736400630884929942272982943613708123011355260915545831043160243523599372006226150289664982113944898886610710824955096724626895416484521819026132177640598691658035986285376355033719094568083122219345722063613609779158338084375331431276527548482566210071347744541292871876134764249704859840950276227627328897424208932988115108907187647698491814375639614313178092528678007370045871748218421786396197284213209022623762734630836006864192414605237248983289006905268988475197599781524158913583701325199090352274252608342971303907669363045656232183978755853064004010895030834921988601355201181158877254807798058635127708445592064519563115094749276606697559529332807221414021024905241788974917755034700510432039890197393691722911126889174394312127254793141624975830429097997705531781908242083922068769027355129212617244130640289994777413026624013157329948333586377955103195844817163822484232700763859290253400376515701986753596890075818544485475785780031843579065754095099970940504640212850809997051128976563880886392410766321449987529690463262182894272302749154535447233331028841215215533602398281107050696017507827602761547816324743297938177204183765821117818869959795031848201322436053103778993541384779857262311465895754085538371969040922420936915076653500310175006188572019017358300979056992161958286882575984331858170857303361269891312794369244896540323192451678830668180455059289743580640736076233561935888109525845803125912388965524166819855977061399043499229843517930169118036812460794615667808961600389778306540324849286501515292799391304510997298128228258006156017389878086272789993321416349205921635696963703558971391123174877353757536774013315034956942784403824181551741629180658414081905650333672638983416786388095026169496605199749691595798835947189777822765198767949699778106683862989103096006505865271003566346191382406011673958404009194852110016915222433459641787170917872140367871023596464051647947388580570774462304347896201676197195521428782313608583714399238092208362933211302942806480175589402387976531080436906856834377344137698180789562645974374155400497754843905032231188252125802180353577510519869570675234892321663406309375
	Command being timed: "python -c print(sum([2 ** x for x in range(20000)]))"
	User time (seconds): 0.69
	System time (seconds): 0.02
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.72
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 34580
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 7646
	Voluntary context switches: 1
	Involuntary context switches: 13
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Use `sort -u` to filter rows that are unique on a particular combination of columns



In [33]:

    
# biggest genome size from each major group?
# cut name, kingdom, group, size
# sort to get biggest first
# sort unique by grou name
# sort again by size
cut -f 1,5,6,7 eukaryotes.tsv | sort -k 4 -r -n | sort -u -k3,3 | sort -k 4 -r -n









    



Pinus_lambertiana	Plants	Land_Plants	27602.7
Homo_sapiens	Animals	Mammals	6503.44
Locusta_migratoria	Animals	Insects	5759.8
Parhyale_hawaiensis	Animals	Other_Animals	4023.76
Salmo_salar	Animals	Fishes	2966.89
Gekko_japonicus	Animals	Reptiles	2490.27
Nanorana_parkeri	Animals	Amphibians	2053.85
Aquila_chrysaetos_canadensis	Animals	Birds	1548.48
Fasciola_hepatica	Animals	Flatworms	1275.11
Symbiodinium_minutum_Mf_1.05b.01	Protists	Other_Protists	609.476
Heligmosomoides_polygyrus	Animals	Roundworms	560.662
Saccharina_japonica	Other	Other	543.426
Cymbomonas_tetramitiformis	Plants	Green_Algae	281.27
Uromyces_viciae-fabae	Fungi	Basidiomycetes	215.71
Blumeria_graminis_f._sp._tritici_96224	Fungi	Ascomycetes	158.945
Rhizophagus_irregularis_DAOM_197198w	Fungi	Other_Fungi	140.696
Sarcocystis_neurona	Protists	Apicomplexans	130.023
Klebsormidium_flaccidum	Plants	Other_Plants	104.21
Trypanosoma_cruzi	Protists	Kinetoplasts	89.9375

Use `column -t` to align output from tabular data



In [34]:

    
# biggest genome size from each major group?
cut -f 1,5,6,7 eukaryotes.tsv | sort -k 4 -r -n | sort -u -k3,3 | sort -k 4 -r -n | column -t









    



Pinus_lambertiana                       Plants    Land_Plants     27602.7
Homo_sapiens                            Animals   Mammals         6503.44
Locusta_migratoria                      Animals   Insects         5759.8
Parhyale_hawaiensis                     Animals   Other_Animals   4023.76
Salmo_salar                             Animals   Fishes          2966.89
Gekko_japonicus                         Animals   Reptiles        2490.27
Nanorana_parkeri                        Animals   Amphibians      2053.85
Aquila_chrysaetos_canadensis            Animals   Birds           1548.48
Fasciola_hepatica                       Animals   Flatworms       1275.11
Symbiodinium_minutum_Mf_1.05b.01        Protists  Other_Protists  609.476
Heligmosomoides_polygyrus               Animals   Roundworms      560.662
Saccharina_japonica                     Other     Other           543.426
Cymbomonas_tetramitiformis              Plants    Green_Algae     281.27
Uromyces_viciae-fabae                   Fungi     Basidiomycetes  215.71
Blumeria_graminis_f._sp._tritici_96224  Fungi     Ascomycetes     158.945
Rhizophagus_irregularis_DAOM_197198w    Fungi     Other_Fungi     140.696
Sarcocystis_neurona                     Protists  Apicomplexans   130.023
Klebsormidium_flaccidum                 Plants    Other_Plants    104.21
Trypanosoma_cruzi                       Protists  Kinetoplasts    89.9375

Use `bashplotlib` to make crude command line charts



In [37]:

    
# first pip install bashplotlib
# histogram of genome size from animals
awk '$5 == "Animals"' eukaryotes.tsv | cut -f 7 | hist -x -b 50









    



 191|  o                                                 
 181|  o                                                 
 171|  o                                                 
 161|  o                                                 
 151|  o                                                 
 141|  oo                                                
 131|  oo                                                
 121|  oo                                                
 111|  oo                                                
 101|  oo                                                
  91|  ooo                                               
  81|  ooo                                               
  71|  ooo                                               
  61|  ooo                                               
  51|  ooo     o                                         
  41|  oooo    o                                         
  31|  oooo o  o           o                             
  21|  oooooooooo        o oooo                          
  11|  oooooooooo o   oooooooooo                         
   1| oooooooooooooooooooooooooooooooo o o         o    o
     ---------------------------------------------------
     0 2 5 7 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5 6 6 
     . 6 2 8 0 3 5 8 0 3 6 8 1 3 6 9 1 4 6 9 2 4 7 9 2 5 
     6 0 0 0 4 0 6 2 8 4 0 6 2 8 4 0 6 2 8 4 0 6 2 8 4 0 
     6 . . . 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 
     2 7 8 9 . . . . . . . . . . . . . . . . . . . . . . 
     5 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 

------------------------------
|          Summary           |
------------------------------
|     observations: 943      |
|    min value: 0.662517     |
|     mean : 1005.694117     |
|   max value: 6503.440000   |
------------------------------

To find wifi cold spots when walking around with your laptop

watch -n 0.2 cat /proc/net/wireless



In [ ]:



In [ ]:



In [ ]:



In [ ]:



In [ ]:

Stuff I learned while writing a Linux course

Codemonkeys 04/08/16

Jupyter with a bash kernel

tree outputs directory structures to the terminal

There's a -t threshold option for du

You can get crude memory benchmarking with /usr/bin/time -v

Use sort -u to filter rows that are unique on a particular combination of columns

Use column -t to align output from tabular data

Use bashplotlib to make crude command line charts

To find wifi cold spots when walking around with your laptop

`tree` outputs directory structures to the terminal

There's a `-t` threshold option for du

You can get crude memory benchmarking with `/usr/bin/time -v`

Use `sort -u` to filter rows that are unique on a particular combination of columns

Use `column -t` to align output from tabular data

Use `bashplotlib` to make crude command line charts